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Perception/Action:  An  Holistic  Approach  II 

John  M.  Flach 
Wright  State  University 

Abstract 

A  general  systems  approach  was  used  to  study  the  emergent  properties  of  the 
human  perception /action  system.  Two  task  domains,  the  control  of  locomotion 
and  the  recognition  of  objects  from  dynamic  occlusion,  were  used  to  study 
human  performance.  The  locomotion  task  involved  the  control  of  altitude. 
Experimental  results  indicate  that  conflicting  results  from  studies  of  altitude 
perception  can  be  explained  when  global  optical  flow  rate  is  considered.  Optical 
analyses  of  the  structure  of  flow  fields,  empirical  results  from  human 
performance  studies,  and  control  theoretical  analyses  of  the  state  space  all 
converge  to  indicate  that  altitude  and  speed  are  not  independent  with  respect  to 
the  problem  of  controlling  locomotion  in  low  altitude  flight.  In  the  dynamic 
occlusion  task,  the  effects  of  time  delays,  visual  noise,  training,  and  instructions 
have  been  evaluated  The  results  tend  to  support  the  hypothesis  that  information 
was  the  critical  determinant  of  performance  in  the  dynamic  occlusion  task.  Mode 
(active  versus  passive  observer)  was  only  important  to  the  extent  that  it  made 
additional  information  available  to  the  observer.  This  conclusion  is  consistent 
with  research  on  adaptation  where  the  "reafference  hypothesis,"  in  which  mode 
played  an  important  role,  is  being  replaced  by  the  "information  hypothesis." 
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Perception/Action:  An  Holistic  Approach  II 

A  fundamental  role  of  the  human  component  in  complex  systems  (e.g., 
vehicular  control,  air  traffic  control,  or  process  control)  is  to  "close-the-loop." 

That  is,  the  humans  are  included  in  the  system  because  of  their  unique  and 
adaptive  abilities  of  perception,  decision  making,  and  motor  control.  Although 
there  have  been  great  advances  in  automated  control  systems,  the  adaptability 
and  generality  of  the  human  have  yet  to  be  matched  by  automated  sensing  and 
control  systems.  This  generality  and  adaptability  of  the  human  controllers  that 
makes  them  attractive  (if  not  essential  in  many  cases)  as  components  within 
complex  systems  poses  a  great  challenge  to  basic  researchers  interested  in 
modeling  human  performance,  as  well  as  for  system  designers  who  need  to  be 
able  to  integrate  across  human  and  electro-mechanical  components  to  predict 
and  evaluate  system  performance.  The  general  goal  of  this  research  program  is 
to  develop  a  framework  for  studying  humans  as  an  adaptive,  closed-loop 
controllers. 

Traditionally,  psychology  has  parsed  the  problem  of  human  performance 
into  problems  of  perception,  cognition,  and  motor  control.  Research  programs 
have  evolved  that  focus  on  one  or  the  other  of  these  components  in  isolation  from 
the  others.  For  example,  those  who  focus  on  perception  often  tightly  constrain 
action  (e.g.,  fixating  the  head  with  bite  boards,  using  brief,  tachistoscopic 
stimulus  presentations,  or  using  restricted  response  protocols  such  as  key 
presses).  Those  who  study  motor  control  go  to  great  lengths  (e.g., 
deafferentation)  to  isolate  motor  control  from  perception.  And  those  who  study 
cognition  select  problems  (e.g.,  tower  of  Hanoi,  missionaries  and  cannibals,  logic 
theorems)  with  minimal  perceptual  and  motor  demands. 

Such  strategies  for  studying  perception  and  action  have  been  successful  in 
reducing  complexity  and  allowing  scientists  to  make  inferences  about  the 
elementary  cognitive  processes  that  combine  to  control  performance.  However, 
these  approaches  miss  the  emergent  properties  that  arise  from  the  coupling  of 
perception  and  action.  Without  an  understanding  of  these  emergent  properties  it 
may  not  be  possible  to  integrate  what  we  have  learned  about  perception  systems 
and  action  systems  in  isolation  into  a  comprehensive  and  general  theory  of 
human  performance  (a  theory  capable  of  guiding  decisions  about  interface 
design  and  training  for  complex  systems  such  as  high  performance  aircraft). 

Thus,  it  is  important  not  to  ignore  these  emergent  properties  of  the  closed-loop, 
perception-action  system.  An  active  psychophysics  (Flach,  1990;  Warren,  1988; 
Warren  &  McMillan,  1984)  is  needed  to  compliment  the  traditional  work  on 
passive  psychophysics,  perception,  and  motor  performance. 

A  fundamental  theme  of  our  research  program  is  that  perception  must  be 
studied  in  the  context  of  action.  The  importance  of  the  coupling  of  perception 
and  action  for  development  of  visually  guided  behaviors  in  cats  was 
demonstrated  in  the  classic  study  of  Held  and  Hein  (1963;  See  also  Held,  1965). 
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In  attacking  the  general  problem  of  active  psychophysics,  our  research  has 
examined  two  classes  of  problems:  the  discrimination  of  objects  using 
information  available  in  dynamic  occlusion  and  the  control  of  locomotion.  The 
first  section  of  this  report  will  discuss  the  problem  of  control  of  locomotion  and 
the  second  section  will  discuss  the  problem  of  dynamic  occlusion. 

Perception  and  Control  of  Altitude 

According  to  Haber  (1987)  jet  fighter  pilots  simply  fly  into  the  ground  in 
visual  flight  conditions  with  no  evidence  of  mechanical  or  operational  failure  or 
medical  emergency  at  a  rate  of  about  two  crashes  per  month.  For  this  reason,  it  is 
critical  that  we  understand  the  information  (or  lack  thereof)  that  pilots  use  to 
control  locomotion.  Our  recent  research  has  focused  on  the  information  in  optic 
flow  for  the  control  of  altitude.  Several  potential  sources  of  information  have 
been  identified.  These  include  splay,  optical  density,  and  depression  angle  (see 
Flach,  Hagen,  &  Larish,  1992;  or  Flach  &  Warren,  in  press,  for  quantitative  and 
graphic  descriptions). 

Recently,  there  has  been  some  controversy  about  the  relative  effectiveness 
of  these  various  sources  of  information.  Research  simulating  fixed  wing  aircraft 
has  consistently  found  splay  to  be  the  most  effective  source  of  information  for 
altitude  (Flach,  Hagen,  &  Larish,  1992;  Warren,  1988;  Wolpert,  1988;  Wolpert  & 
Owen,  1985;  Wopert,  Owen,  &  Warren,  1983).  Research  simulating  rotary  wing 
aircraft  has  consistently  found  depression  angle  to  be  the  most  effective  source  of 
information  for  altitude  (Johnson,  Bennett,  O'Donnell  &  Phatak,  1988;  Johnson, 
Tsang,  Bennett,  &  Phatak,  1989). 

In  a  recent  study  in  our  lab  (Kelly,  1993;  Kelly,  Flach,  Garness  &  Warren, 
1993),  we  looked  at  global  optical  flow  (GOF)  rate  as  a  possible  intervening 
variable.  The  hypothesis  was  that  when  a  vehicle  is  in  hover  or  moving  at  very 
low  forward  speeds  (low  GOF  rate),  depression  angle  would  be  most 
informative.  However,  when  a  vehicle  is  moving  forward  at  higher  speeds  (high 
GOF  rate),  then  splay  is  most  informative.  To  test  this  hypothesis,  four  flow 
textures  (splay  only  (vert),  depression  angle  only  (horz),  block,  and  dot  [in 
principle  grid  and  dot  texture  should  contain  both  splay  and  depression 
information]  were  crossed  with  four  global  optical  flow  rates  (0,  0.25, 1, 4). 

Results  showed  an  interaction  as  shown  in  Figure  1.  Performance  with  splay 
only  (vert),  dot,  and  block  textures  was  independent  of  GOF  rate  and  had  lower 
error  rates  than  with  depression  (horz)  texture.  Performance  was  generally  poor 
with  horizontal  texture  and  became  worse  at  the  highest  flow  rate.  In  no  case, 
not  even  at  the  hover  (0  GOF  rate),  condition  was  depression  angle  superior. 
Thus,  we  were  unable  to  replicate  the  results  of  Johnson  et  al. 

A  second  study  has  just  been  completed.  In  the  previous  study  GOF  rate 
was  manipulated  within  subjects.  In  the  second  study,  we  tested  subjects  only  in 
a  hover  condition.  Under  that  condition  we  were  able  to  replicate  the  results  of 
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Figure  1.  RMS  Altitude  error  as  a  function  of  texture  and  global  optical  flow  rate. 


Rate  ol  Chan gs  of  Attitude 


Figure  2.  A  state  space  representation  showing  altitude  and  rate  of  change  of 
altitude.  The  curves  in  this  diagram  represent  paths  through  the  state  space.  The 
dark  curves  (a,  c,  d,  &  e)  represent  critical  boundaries  for  different  types  of  low 
level  flight.  The  space  outside  of  the  areas  bounded  by  these  critical  limits 
represent  conditions  where  there  is  no  control  action  that  will  avoid  crashing  into 
the  ground  or  ceiling  that  delimits  safe  areas  within  the  state  space.  Curves  b 
and  f  are  examples  of  paths  where  crashing  into  the  ground  or  ceiling  is 
unavoidable.  The  hypothesis  is  that  safe  flight  requires  that  the  pilot  can 
perceive  position  with  respect  to  these  boundaries. 
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Johnson  et  al.  We  found  that  depression  angle  resulted  in  superior  performance. 
Splay  angle  resulted  in  the  poorest  performance.  These  results  suggest  that  splay 
is  generally  better  (over  a  wide  range  of  GOF  rates),  but  that  depression  angle  is 
better  for  the  specific  case  of  0  GOF  (hover).  More  importantly,  this  program  of 
research  indicates  that  altitude  and  speed  may  not  be  functionally  independent 
sources  of  information.  This  same  conclusion  has  been  suggested  from  analytical 
analysis  of  the  control  or  state  space  for  vehicular  control  (Flach  &  Warren,  in 
press).  Figure  2  illustrates  that  critical  boundaries  for  low  altitude  flight  are 
generally  specified  as  a  joint  function  of  altidude  and  rate  of  loss  of  altitude.  This 
suggests  that  it  will  be  important  to  examine  higher  order  structures  in  optic  flow 
that  are  invariant  over  variations  in  altitude  and  speed  (e.g.,  tau  or  time-to- 
contact). 


Dynamic  Occlusion 

The  dynamic  occlusion  task  required  subjects  to  identify  wire  frame,  three 
dimensional  objects  using  the  information  made  available  from  the  temporal- 
spatial  pattern  of  occlusion  and  disocclusion  created  as  these  objects  are  moved 
within  a  field  of  point  lights.  This  task  has  been  explained  in  a  previous  report  to 
AFOSR  (Flach  1992;  see  also  Flach,  Allen,  Brickman,  &  Hutton,  1992).  This  task 
has  been  used  to  evaluate  performance  as  a  function  of  the  mode  by  which  the 
pattern  of  occlusion  is  generated.  One  dimension  of  mode  is  whether  the  subject 
is  active  or  passive.  Subjects  have  been  run  in  yoked  pairs.  One  subject  in  each 
pair  was  able  to  control  the  pattern  of  occlusion/disocclusion  using  a  joystick 
control.  This  is  the  "active"  subject.  The  second  "passive"  subject  saw  the  same 
pattern  of  occlusion/disocclusion,  but  had  no  control  of  the  motion  of  the  object. 
Another  dimension  of  mode  is  whetner  the  pattern  of  occlusion/ disocclusion  is 
created  by  a  subject  who  has  the  intention  of  identifying  the  object  (subject 
generated  information)  or  whether  the  pattern  was  generated  in  a  stereotypical 
pattern  by  the  computer  (e.g.,  constant  rotation  at  fixed  velocity). 

Previous  research  with  the  dynamic  occlusion  paradigm  has  indicated  that 
under  the  yoked  conditions  in  which  the  information  (pattern  of  dynamic 
occlusion)  was  identical  for  active  and  passive  observers,  there  was  no  difference 
in  performance  (Flach,  1992;  Flach  et  al.  1992).  Performance  was  equivalent  for 
the  two  classes  of  observers.  Further,  an  interaction  with  practice  has  been  found 
for  the  other  dimension  of  observation  mode.  Early  in  practice  subjects  perform 
better  with  computer  generated  information  than  with  subject  generated 
information.  With  practice,  however,  the  performance  of  these  two  groups 
converged. 

Over  the  last  year,  further  experiments  have  examined  the  effects  of  noise, 
mode  awareness,  time  delay,  training,  and  shape  in  the  dynamic  occlusion 
paradigm. 
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Noise.  In  previous  research,  the  only  variation  visible  in  the  display  was  the 
result  of  motion  of  the  object.  When  an  edge  of  the  object  was  positioned  in  front 
of  a  point  light,  the  light  disappeared,  and  when  the  edged  moved  from  in  front 
of  the  point  light  it  reappeared.  Noise  was  added  to  the  display  so  that  the 
points  would  appear  and  disappear  at  random  intervals,  independent  of  the 
position  of  the  object.  Thus,  instead  of  being  positioned  in  a  field  of  steady  point 
lights,  the  object  was  now  positioned  in  a  field  of  randomly  blinking  point  lights. 
The  rationale  for  this  new  dimension  was  the  hypothesis  that  the  correlation 
between  intended  actions  and  patterns  of  occlusion  would  allow  the  "active" 
subject  to  distinguish  between  signal  (occlusions  resulting  from  object  motion) 
and  noise  (random  blinking  of  point  lights).  In  turn,  it  was  thought  that  this 
would  give  the  active  subject  an  advantage  in  recognizing  the  objects. 

Experiments  conducted  with  noise  found  tentative  support  for  our 
hypothesis.  First,  the  experiments  replicated  previous  results  showing  no  effect 
of  mode  (active  vs.  passive)  in  no  noise  conditions.  Performance  levels  were 
reduced  when  noise  was  introduced.  When  noise  was  manipulated  within 
subjects  a  significant  mode  by  noise  interaction  was  obtained.  Active  and 
passive  observers  were  equivalent  when  no  noise  was  present,  but  subjects 
where  better  in  the  active  mode,  when  noise  was  included  (see  Figure  3).  A 
similar  pattern  was  found  in  a  second  experiment  in  which  noise  was 
manipulated  between  subjects.  However,  the  interaction  was  not  statistically 
significant  in  the  second  experiment. 

Mode  Awareness.  In  all  our  previous  experiments,  the  subjects  have  had  explicit 
awareness  of  the  mode  in  which  they  were  operating.  That  is  passive  subjects, 
knew  that  they  could  not  control  the  motion  of  the  object  and  active  subjects 
knew  that  they  could  control  the  object.  Even  if  this  were  not  made  explicit  by 
instructions  —  when  no  noise  was  present,  it  would  be  easy  for  the  subjects  to 
quickly  discover  the  constraints  on  their  particular  mode.  However,  the  addition 
of  noise  in  the  display  opens  the  possibility  to  manipulate  awareness.  An 
experiment  was  run  in  which  all  subjects  were  instructed  that  they  had  control 
over  the  object.  In  fact,  however,  they  only  had  control  on  a  randomly  selected, 
half  of  the  trials  within  a  block.  Thus,  on  half  the  trials  one  subject  had  control 
while  the  second  subject  was  passive.  On  the  other  half,  the  roles  were  reversed. 
However,  the  key  difference  between  previous  studies  was  that  the  subjects 
believed  that  they  had  control  on  every  trial.  Thus,  their  actions  were 
"superstitious"  on  half  of  the  trials. 

When  subjects  were  run  in  this  ambiguous  mode  condition,  there  was  a 
substantial  effect  of  mode,  with  "true"  active  subjects  performing  better  than 
"superstitious"  active  subjects. 

Time  Delay.  A  critical  parameter  of  any  closed-loop  control  system  is  the  time 
delay.  Thus,  it  is  likely  that  this  will  be  an  important  factor  to  consider  for 
evaluating  the  coupling  of  perception  and  action.  We  examined  two  levels  of 
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Figure  3.  Interaction  between  noise  and  mode.  For  Noise  =  1  no  random 
blinking  was  present  in  the  display.  For  Noise  =  2  random  blinking  was  present. 
Solid  symbols  represent  Active  subjects.  Open  symbols  represent  Passive 
subjects. 
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time  delay  (200  ms  and  2  s)  with  or  without  noise.  Results  showed  no  effect  of 
the  differential  delays.  The  noise,  again,  resulted  in  a  significant  decrement  in 
performance.  No  interactions  with  mode  (active  vs.  passive)  w’ere  found.  We 
have  some  tentative  hypotheses  about  why  time  delay  was  not  effective  for  this 
task.  However,  further  studies  are  required. 

Training.  In  earlier  studies,  an  interaction  was  found  between  practice  and  the 
source  of  motion  (subject  generated  vs.  computer  generated).  Early  in  practice 
performance  is  best  with  the  stereotypic  patterns  generated  by  the  computer  than 
with  the  exploratory  motions  generated  by  the  subjects.  Later  in  practice  the 
performance  for  the  two  types  of  motion  converges.  One  hypothesis  for  this 
pattern  of  results  is  that  subjects  are  learning  to  use  the  joystick  early  in  practice 
and  are  able  to  devote  less  attention  to  the  task  of  identifying  the  object.  Later,  as 
they  become  more  skilled  at  manipulating  the  object  using  the  joystick, 
performance  improves  to  a  level  equivalent  to  that  obtained  when  the  computer 
generates  the  motion.  To  test  this  hypothesis,  a  training  manipulation  was 
tested.  Subjects  were  trained  to  move  "visible"  3-d  objects  with  the  joystick  prior 
to  being  tested  in  the  dynamic  occlusion  task.  However,  the  training  to 
familiarize  subjects  with  the  mapping  from  the  joystick  to  object  motion  did  not 
change  the  pattern  of  results.  Thus,  the  unfamiliar  mapping  of  the  stick  to  object 
motion  does  not  appear  to  explain  the  early  differences  between  subject 
generated  and  computer  generated  motions. 

Shape.  A  final  manipulation  examined  the  shape  of  the  objects  used.  In  all  the 
previous  studies  six  3-d  shapes  were  used.  These  shapes  may  be  discriminated 
based  on  gross  features  (e.g.,  short  and  fat  versus  tall  and  narrow).  Thus, 
subjects  may  not  be  required  to  identify  the  shape  per  se,  but  merely  discriminate 
between  it  and  the  other  five  objects  based  on  these  gross  features.  Perhaps  if  the 
distinctions  between  the  objects  were  more  subtle,  then  there  may  be  some 
advantage  for  the  active  subject.  To  test  this  hypothesis  a  new  object  set  was 
constructed.  This  set  included  squashed  and  stretched  versions  of  the  previous 
shapes.  The  result  of  changing  the  dimensions  along  different  axes  was  to  make 
it  more  difficult  to  discriminate  among  the  shapes.  Results  showed  that  this 
manipulation  made  the  task  significantly  more  difficult.  However,  performance 
for  active  and  passive  observers  remained  t  quivalent  with  this  new  object  set. 

Summary.  The  results  of  the  new  manipulations  are  all  consistent  with  an 
"information  hypothesis"  regarding  the  coupling  of  perception  and  action 
(Welch,  1986).  This  hypothesis  suggests  that  mode  per  se  (e.g.,  active  vs.  passive 
observer)  is  not  the  critical  dimension.  Rather,  performance  depends  on  the 
quality  of  information  available  to  the  observer.  The  active  observer  will  perform 
better  than  the  passive  observer,  if  and  only  if,  the  active  mode  makes 
information  available  that  is  not  available  for  the  passive  observer.  In  our 
paradigm  this  happens  when  background  noise  is  present.  The  active  observer 
has  knowledge  of  the  commanded  motio.'  and  thus  is  better  able  to  distinguish 
the  signal  (occlusion  resulting  from  object  motion)  from  noise  (random  blinking). 
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Also,  when  the  mapping  from  the  control  stick  to  the  object  is  "real"  versus 
"superstitious,"  then  there  is  an  advantage  for  the  active  observer.  In  all  other 
conditions  that  have  been  tested,  performance  was  equivalent  for  active  and 
passive  observers. 


General  Conclusions 

A  common  theme  has  emerged  from  both  research  paradigms  (altitude 
control  and  dynamic  occlusion).  Both  paradigms  highlight  the  importance  of 
information  in  linking  perception  and  action.  The  ability  to  make 
discriminations  of  either  altitude  or  object  shape  depends  on  the  ability  to 
differentiate  variation  within  the  optic  array  that  are  specific  to  the  critical  task 
dimension  from  other  optical  variations  within  the  flow  field.  For  the  pick-up  of 
altitude,  variations  from  global  optical  flow  make  it  difficult  to  judge  altitude 
changes  using  horizontal  texture  (depression  angle).  This  noise  is  minimal  in  the 
hover  condition  and  increases  with  increasing  optical  flow  rate.  Thus,  we  see 
differences  in  subjects'  ability  to  control  altitude  using  horizontal  texture  as 
global  optical  flow  rate  is  changed.  In  the  dynamic  occlusion  task,  differences 
were  found  between  active  anu  passive  observers  only  when  the  mode  provided 
differential  access  to  information.  This  was  seen  in  noise  conditions  where 
knowledge  of  the  commanded  motion  was  available  to  the  active  subject. 
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